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APPARATUS, SYSTEMS AiND METHODS 
FOR CONTROLLING GRAPHICS AND 
VIDEO DATA IN MULTLMEDU DATA 

PROCESSING AND DISPLAY SYSTEMS 

5 

TECHNICAL FIELD OF THE INVENTION 

The present invention relates in general to muitimedia 
processing and display systems and in particular to appara- 
tus, systems and methods for controlling graphics and video lo 
data overiay in muUimedia processing and display systems. 

CROSS-REFERENCE TO RELATED 
APPLICATIONS 

The following copending and coassigned United States 
paient appiicaiions contain related information and are 
incorporated herein by reference: 

U.S, patent application Ser. No. 08/098.846 (Attorney's 
Docket No. PaSIO-PllUS), entitled "System And Method 20 
For The Mixing Of Graphics And Video Signals/* and filed 
Jul 29, 1993; and 

U.S. patent applicauon Ser. No. 08/223,845 (Attorney's 
Docket No. P3510-P21US), entided "Apparatus, Systems 
And Methods For Processing Video Data In Conjunction 25 
With A Multi-Format Frame Buflfer," and filed Apr, 6, 1994. 

BACKGROUND OF THE INVENTION 

As multimedia information processing systems increase 
in popularity, system designers must consider new tech- 
niques for controlling die processing and display of data 
simultaneously generated by multiple sources. In particular, 
there has been substantial demand for processing systems 
which have the capability of concuircntly displaying both 
video and graphics data on a single display screen. The 
development of such systems presents a number of design 
challenges, not only because the format differences between 
graphics and video data must be accounted for, but also 
because of end user driven requirements that these systems ^ 
allow for flexible manipulation of the data on the display 
screen. 

One particular techni<^e for simultaneously displaying 
video and graphics data on a single display screen involves 
the generation of "windows." In this case, a stream of data 45 
from a selected source is used to generate a display within 
a particular region or "window" of the display screen to the 
exclusion of any non-selected data streams defining a dis- 
play or pan of a display conespooding to the sanoe region of 
the screen. The selected data stream generating the display so 
window "ovcriays** or "ocdudcs" the data &om the nonsc- 
lected data streams which lie *1}ehind** the displayed data. In 
one instance, die overall content and appearance of the 
display screen is defined by graphics data and one or more 
"video windows" generated by data &om a video source 55 
occlude a corresponding region of diat graphics data. In 
other instances, a video display or window may be occluded 
or overlaid by graphics data or even another video window. 

In the muidmedia environment, the '^windowing" 
described above yields substantial advantages* Among other 60 
things, the user can typically change the size and location on 
the display screen of a given window to flexibly manipulate 
the content and appearance of the data being displayed. For 
example, in the case of combined graphics and video, the 
user can advantageously create custom composite visual 65 
displays by combining multiple video and graphics data 
streams in windowing environment 
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In order to efBcientiy control windows in a multimedia 
environment efficient frame buffer management is required. 
Specifically, a frame buffer control scheme must be devel- 
oped which allows for the efi5cient storage and retrieval of 
5 multiple types of data, such as video data and graphics data. 
To be cost competitive as well as functionally ef&cient, such 
a scheme should minimize the number of memory devices 
and the amount of control circuitry required and should 
insure that data flow to the display is subjected to minimal 
10 delay notwithstanding data type. 

One of the major difficulties in managing video in a 
combined video and graphics windowing environment 
results from the fact that the video data being received and 
displayed are constantly being updated, typically at a rate of 
15 thirty frames per second In contrast, the graphics data are 
normally generated once to define the graphics display and 
then remain static until the system CPU change that graphics 
display. Thus, the occlusion (overlay) of video data with 
graphics data requires that the static graphics data "in front 
20 of* the video data not be destroyed each time the video 
window is updated. A second concern with windowing 
systems operating on both video and graphics data is the 
formatting differences between the video and graphics data 
themselves since video is ^icaliy '^'^^ a YUV 

^ color space while graphics is digitized into an RGB color 
space. Hencct any coinbination video and graphics window- 
ing system must have the capability of effidently handling 
data within both the YUV and RGB formats. 

Thus, due to the advantages of windowingt the need has 
arisen for efficient and cost effective windowing control 
circuitry. Such windowing circuitry should allow for the 
simultaneous processing of data received firom multiple 
sources and in multiple formats. In particular, such window- 
ing control circuitry should be capable of effidently and 
inexpensively controlling the occlusion and/or overlay of 
video and graphics data in a windowing cnvironmem. 



SUMMARY OF THE INVENTION 

^ The prindples of the present invention in general provide 
for the flexible control of graphics and video data in a 
display control environment In particular, an entire frame of 
video jatg, graphics data, or a combination of both, may be 

^5 stored in on-screen memory and rasteted out with the 
generation of the corresponding display screen. A windo w of 
graphics or video data can then be stored in off-screen 
memory and retrieved when the raster scan generating the 
display reaches the desired position on the display for the 

CQ video window. The window of data ftom off-screen memory 
c an then be ovcrlaycd over the data being rastercd out of the 
on-screen memory under one of three conditions. In a first 
mode, pixds from the off-screen memory are rastercd only 
when the taster scan has reached the position on the display 

55 selected for the window. In a second mode, a window of data 
is rastered from the off-screen memory when the display 
raster scan has reached the display window position and 
graphics data being rastered from the on-screen memoty 
matches a color key. In a third mode, the window data is 

60 rastered out of the off-screen memory when the data being 
output from the on-screen memory matches the color key, 
notwithstanding the position of the raster scan. 

According to a first embodiment of the present invention, 
a graphics and video controUcr is provided which includes 

65 a dual aperture interface, each word associated with an 
address to a selected one of on-screen and off-screen areas 
of an associated unified frame buffer as either graphics or 
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video pixei data. Circuiiry is provided for writing a word of 
the pixel data received by the inierfacc to a one of ihe 
on-screen and off- screen memory areas corresponding to the 
address associated with the received word. Circuitry is also 
included for selectively retrieving graphics and video data 5 
from the on-screen and off- screen memory areas. A first 
pipeline is provided for processing graphics data retneved 
from Che frame buffer and a second pipeline is provided for 
\ideo processing data retrieved from the frame buffer. 

According to a second embodiment of the present inven- iq 
lion, a controller is provided which includes a dual apeaure 
port for receiving video and graphics data, each word of the 
data received with an address associated directing the word 
to be processed as either graphics or video data and off- 
screen memory spaces of a frame buffer. A second port is 
included for receiving rea]-time video data. Circuitry is 
provided for generating an address associated with a 
selected one of the memory spaces for each word of received, 
real-time video data. Circuitry is included for writing selec- 
tively the words into the on-screen and off-screen memory 
spaces of the frame buffer. Circuitry is also provided for 
selectively retrieving the words of data from the on-screen 
and off-screen spaces as data is rastered for driving a display. 
A graphics backend pipeline processes ones of the graphics 
words of data remevcd from the frame buffer. A video ^ 
backend pipeline is provided for processing ones of the 
video words of data retrieved firom the frame buffer, the 
circuitry for retrieving always rastering a stream of graphics 
data from the frame buffer to the graphics pipeline and 
rastering video data to the video backend pipeline when a 
display raster scan reaches a display position of a video 
window. An output selector is included for selecting for 
output between words of data output from the graphics 
backend pipeline and words of data output from the video 
backend pipeline. 35 
According to 

a third embodiment of the present invention, a 
display system is provided which includes first and second 
parallel backend pipelines. A multi-fonnat frame buffer 
memory is included having on-screen and off-screen memo- 
ries each operable to simultaneously store data in graphics 40 
and video formats. A dual aperture port is provided for 
receiving both graphics and video data as directed by an 
address associated with each word of data received. Cir- 
cuitry for writing is included for writing a word of video or 
graphics data into a selected one of the on-screen and 45 
off-screen areas of the multi-frame buffer: Memory control 
circuitry controls the transfer of data between the first and 
second [back-end] backend 

pipelines and the frame buffer. The system 
further includes a display unit and overiay control circuitry 
for selecting for output to the display unit between data so 
provicted by the first backend pipeline and data provided by 
the second backend pipeline. 

A fourth embodimeia of the present invention comprises 
a display data processing system which includes circuitry for 
writing data into an on-scrcen space of a frame buffer and S5 
circuitry for writing data into an off-screen space of the 
frame buffer. A video pipeline is provided for processing 
video data output from a selected one of the on-screen and 
off-screen spaces. The video pipeline includes a first first- 
in/first-out memory for receiving selected pixel data from 60 
the selected space. The video pipeline also includes a second 
first-in/first-out memory disposed in parallel to the first 
first-in/first-out memory for receiving other selected data 
from the selected space in the frame buffer. An interpolator 
is provided as part of the video pipeline for generating 6S 
additional data by interpolating data output from the first and 
second first-in/first-out memories, A graphics pipeline is 
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disposed in parallel to the video pipeline for processing 
graphics data output from a selected one of the on-screen 
and off-screen spaces. Finally, an output selector is provided 
for selecting between data output from the video pipeline 

5 and data output from the graphics pipeline. 

The principles of the present invention allow for the 
construction of circuits and systenis with substantial advan- 
tages over the prior art. Among other things, the principles 
of the present invention allow both graphics and video data 

^0 to be stored in a single unified frame buffer and retrieved 
therefrom in a number of different ways. For example, a 
combination of graphics and video daia may be stored in the 
on-screen memory and simply rastered out during screen 
refresh. In another case, an entire screen of graphics or video 

15 data may be stored in the on-screen memory while a window 
of gr^hics or video data is stored in the off-screen portion 
of memory. The window data can then be rastered out to 
selectively overlay a portion of the data being rastered out of 
the on-screen memory. The overlay may be controlled by 

20 either window display position with a match of the on- 
screen data being rastered out and a color key, or both. 

The embodiments of the present invention provide for the 
eflScient and inexpensive overlay of video and graphics data 
in a windowing environment In particular, the use of color 
comparison to determine the overlay of data in a window 
region eliminates the need for precise x- and y-position Ha^^^ 
for the location of that window and allows for video crop- 
ping to be performed Further, the use of graphics data to 
control overiay provides substantial advantages in thai 
graphics data is less subject to the graininess and noise 
problems often found with video data Further, the user is 
given total control of overlay operations when keying on 
graphics data because the graphics data is computer gener- 
ated, whereas the video data is o^tured data. 

BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present inven- 
tion, and the advantages thereof, reference is now made to 
40 the following descriptions taken in conjunction with the 
accompanying drawings, in which: 

FIG. 1 is a top level functional block diagram of a 
multi-media processing and display system embodying the 
principles of the present invention; 

FIG. 2 is a moxe detailed functional block diagram of the 
VGA controllff depicted in FIG. 1; 

FIG. 3 is an expanded functional block diagram of por- 
tions of the controller of FIG. 2 with emphasis on the overlay 
5Q control features; 

FIG. 4A is a detailed functional block diagram of a first 
embodiment of the color comparison circuitry of FIG. 3; 

FIG. 4B is a detailed functional block diagram of a second 
embodiment of the color comparison circuitry of FIG. 3; and 

FIG. 5 is a detailed functional block diagram of a selected 
one of the video window position control circuits depicted in 
HG. 3, 



DESCRIFnON OF THE INVENTION 

60 

FIG. 1 is a high level functional block diagram of a 
multi-media processing and display system 100 operable to 
process and simultaneously display on a single display 
screen both graphics and video data according to the prin- 
65 ciples of the present inventioiL Display system 100 includes 
a central processing unit (CPU) 101 which controls the 
overall operation of system 100 and generates graphics data 
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defining graphics images co be displayed. CPU 101 com- 
municaics wiih the remainder of the system discussed beiow 
via a local bus 103. System 100 also includes a real-time 
video data source 104. A real lime video stream may be 
presented to the system VGA conlroUer 105 in one of two 3 
ways. First, video data source 104 may be coupled to local 
bus 103 and a video data stream introduced through dual 
aperture addresses. In this case, video source 104 will 
directly address the system frame buffer 107. Second, video 
source 104 may be coupled directly to VGA controller 105 
via a dedicated bus 109 or "video port." In this instance, 
VGA conu-oller 105 generates the required addresses into 
frame buffer 107. Real-time video source 104 may be, for 
example, a CD ROM unit, a laser disk unit, a videotape unit, 
television cable outlet or other video data source outputting 
video data in a YUV format. CPU 101 operates in conjunc- 
tion with a system memory 108 which stores graphics and 
video data on a real-time basis. System memory 108 may be 
for example random access memory (RAM), floppy disk, 
hard disk or other type of storage device. 

A VGA controller 105 embodying the principles of the 
present invention is also coupled to local bus 103. VGA 
controller 105 will be discussed in detail below; however, 
VGA controller 105 generally interfaces CPU 101 and video 
source 104 with a display unit 106 and a multiformat system 
frame buffer 107. Frame buffer memory 107 provides tem- 
poraiy storage of the graphics and video data during pro- 
cessing prior to display on display unit 106. According to the 
principles of the present invention, VGA controller is oper- 
able in selected modes to store graphics and video data 
together in frame buffer 107 in their native formats. In a 
preferred embodiment, the frams buffer area is partitioned 
into on-screen memory and off-screen memory. Frame 
buffer 107 is also a "unified" memory in which video or 
graphics data can be scored in either the on-screen or 
off-screen areas. In the preferred embodiment, display unit 

106 is a conventional raster scan display device and ftame 
buffer 107 is constructed from dynamic random access 
memory devices (DRAMs). 

FIG. 2 is a more detailed functional block diagram of ^ 
VGA controller 105. The priniary circuitry blocks of VGA 
controller 105 include video front-end video pipeline 200, 
memory (frame buffer) control circuitry 201, CRT/window 
control circuitry 202, video window control registers 209i 
video [back-end] backend pipeline 204 and graphics 

fback-end] haclcend pipeline 204 and graphics back-end pipeline 
205. VGA controller 105 further includes a CPU interface 
206 for exchanging instructions and data via a PCI or VL 
bus, such as local bus 103 in system 100, with CPU 101. A 
write buffer 207 and conventi<ma2 graphics controller 208 
allow CPU 101 to ditectiy control data within &ame buffer ^ 

107 via memory control dicuitcy 201. 

In the preferred embodiment of system 100, CPU 101 can 
write video daia and/or read and write graphics data to frame 
buffer 107 via CPU interface 206. In particular, CPU 101 can 
direct each pixel to the frame buffer usiog one of two maps 55 
depending on whether that pixel is a video pixel or a 
graphics pixel In the preferred embodiment, each word of 
pixel dau (''pixel") is associated with one of two addresses, 
one which directs interpolation of the pixel as a video pixel 
through video front-end pii»line 200 and the other which ^ 
direcu interpolation of the pixel as a graphics pixel through 
write buffer 207 and graphics controller 208. As a conse- 
quence, either video or graphics pixel data can then be input 
to CPU interface 206 from the PCIA^ bus through a single 
''dual aperture" port as a function of the selected address. 6S 

Data which is input through the video port 211 is 

address- 
free. In this case, video vnndow controls 213 generates the 



required addresses to either the on-screen memory area or 
the o6f-screen memory as a function of display location for 
the video window. In the preferred embodiment, window 
controls 213 generate addresses using the same video con- 
5 irol registers 203 used to control retrieval of the video in the 
[back-end] backend pipeline (i.e., the screen x and y position registers 
500 and 501 discussed below in conjunction with FIG. 5). 
When data is being received through both the CPU interface 
206 and the VPORT 211 simultaneously, the data is inter- 
leaved into memory with the two write buffers 207 and 217 
buffering the data such that neither stream is interrupted or 
forced into a wait state at the source component (i.e., bus 103 
or video source 104). 

It should be noted at this point that frame buffer 107 
includes at least two different data areas or spaces to which 
data can be directed bv the given address (either CPU 103 
or controls 213 generated.) 

Each space can simultaneously 
store graphics or video data depending on the selected 
display configuratioiL The on-screen area corresponds to the 
display screen; each pixel rastered out of a given pixel 
location in the on-screen area defines a corresponding screen 
pixel. The off-screen area is used to store data defining a 
window for selectively overiaying the data from the on- 
screen memoiy, fonts and other data necessary bv controller 
25 105. Further, as will be discussed [fiirther] below, both graphics 
and video data may be rastered £rom &ame buffer 107 and 
passed through video backend pipeline 204 while only 
graphics data is ever passed through graphics backend 
pipeline 205. 

According to the principles of the present invention, there 
are alternate ways of storing and retrieving graphics and 
video data from unified frame buffer 107. 

For example. CPU 103 may write a static graphics back- 
ground into part of the on-screen memory with the remain- 

35 ing "window** in the on-screen memory area filled with 
playback video data. 'TIayback" video data can be either (1 ) 
live video data input from the VPOilT; (2) YUV (video) data 
written through interface 206 fay CPU 103; or (3) tme color 
(5:5:5, 5:6:5, or 8:8:8) RGB graphics data (for example 

40 animation graphics data) written in through eitiier the 
VPORT or interface 206. Similariy, a playback video back- 
ground and a window of graphics data may be written into 
the on-screen area. In each of these cases, the data is rastered 
out as the display is without overiav: the video playfaack data 

45 is passed throuj^i [the video playback data is passed through] 
the video backend pipeline 204 as a function of display 
position by controls 202 and the gn^hics data passed 
through die graphics backend pipeline 250* 
Windows of data retrieved from the off-screen memory 

50 can be retrieved and used to occlude a portion of the data 
being rastered out of the on-screen mem<»y. For example, a 
window of playback data can be stored in the off-screen 
menoory and a frame of static graphics data (either true color 
data or indices to GLUT 234) stored in the on-screen 

55 menKKy. In this case, the static gr^Mcs are rastered out of 
the on-screen memory without Intermption and passed 
through the graphics backend pipeline 205. The window of 
data in the off-screen memory is rastered out only when the 
display position for the window has been reached .by the 

60 display rasier and is passed Uirough video backend inpeiine 
204. As discussed below, data from the video backend 
pipeline 204 can then be used to selectively occlude (over- 
lay) the data being ou^ fmm the graphics backend pipe- 
line 205. A window of static graphics data (true color or 

65 indices to the GLUT 234) can be stored in off-screen 
memory and used [of] to , , ^ 

overlay playback video from die 

on-screen memory[, the]. The 

playback video data is passed 
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through the video backcnd pipeline 204 and ihe window of 
sialic graphics daia is passed through the graphics backend 
pipeline 205, 

Bit bloclc transfer (BitBLT) circuitry 209 is provided to 
ailow biocks of graphics data within frame buffer 107 to be 5 
iransferred, such as when a window of graphics data is 
moved on the display screen by a mouse. Digital-to-analog 
convener (DAC) circuitry 210 provides the requisite analog 
signals for dnvmg display 106 in response to the receipt of 
cither video data from video [back-end] backend 

pipeline 204 or jq 
graphics data from [back-end pipe Une] backend pipeline 

208. 

(n impiemcnting the operations discussed above, video 
front-end pipeline 200 can receive data from two mutually 
exclusive input paths. First, in the "playback mode»'* play- 
back (non-real time) data may be received via the PCI bus 15 
through CPU interface 206, Second, in the "overlay emu- 
lation mode" cither real-time or playback video may be 
received through the video port interface 211 (in system 100 
video port interface 211 is coupled to bus 109 when real- 
time data is being received). The selection of video from the 20 
PCI bus or video from video port interface 211 is controlled 
by a multiplexer 212 under the control of bits stored in a 
video front-end pipeline control register within video control 
registers 203. In the playback mode, either CPU 101 or a 
PCI bus master controlling the PQ bus provides the frame 25 
buffer addresses allowing video front-end pipeline 200 to 
map data into the frzme buScr separate and apart from the 
graphics data. In the overlay emuladoc mode* overlay input 
window controls 213 receives framing signals such as 
VSYNC and HSYNC, tracks these sync signals with 30 
counters to detennine the start of each new frame and each 
new line, generates the required addresses for the real-time 
video to the frame buffer space using video window position 
data received from window controls 222 (as discussed 
above, in the preferred embodiment, video data is always 33 
retrieved from either the on-screen on off-screen memory 
and passed through video back-end pipeline 204 as a func- 
tion of display position) and thus the position data from 
controls 222 is used to both write data to memory and 
retrieve data therefromDJ.In general, overlay input video 40 
control windows are controlled bv the same registers which 
control the [back-end] backend 

video pipeline 204, although the req- 
uisite counters and comparators are located internal to 
overlay input video control drcuitty 213. 

Video front-end pipeline 200 also includes encoding 45 
circuitry 214 that is operable to truncate 16-bit YUV 422 
data into an 8-bit formal and then pack four such 8-bit 
encoded words into a single 32-bit word which is then 
written into the video frame buffer space of &aine buffer 
105. Conversion dicuxtty 215 is operable to conven RCB so 
555 data received from either the CPU interface 206 and the 
PCI bus or VPORT W 211 into YCrCb (YUV) data prior to 
encoding by encoding circuitry 214. Convenion circuitry 
215 allows graphics data (for example in a 5:5:5 or 5:6:5 
format) to be introduced through the VPORT or graphics 55 
data to be converted, packed and stored in a YUV format in 
the off-screen menwiy space by CPU 101, For a more 
complete description of encoder 214 and the associated 
decoder 225 of video pipeline 204, reference is now made to 
incorporated copending coassigned applicadon Ser. No. 60 
08/223,845. The selection and control of the encoding 
circuitry 214 and conversion circuitry 215 is implemented 
through multiplexing circuitnes 212 and 216, each of which 
are controlled by bits in the video control registers. Hnaliy, 
video front-end pipeline 200 includes a write buffer/FIFO 65 
217 which in one embodiment acts as a write buffer and in 
an alternate embodiment acts as a FIFO for the video 
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backcnd pipeline 204. In embodiments where buffer 217 acts 

as a write buffer for then zooming on the [back-end,] backend^ as 

discussed below is by replication. In embodiments where 

buffer 217 operates as a FIFO, then the VPORT and from 

and end color conversion by converter circuitry 215 are not 

used for writing data to frame buffer 107. 

Memory control circuitry 201 includes an arbiter 218 and 
a memory mterface 219, Arbiter 218 prioritizes and 
sequences requests for access to frame buffer 107 received 
from video front-end pipeline 200, graphics controller 208 
and bit block transfer circuitry 209. Arbiter 218 further 
sequences each of these requests with the refresh of the 
display screen of display 106 under the control of CRT 
controller 202. Memory interface 219 controls the exchange 
of addresses, data, and control signals (such as RAS, CAS 
and read/write enable) to and from frame buffer 107. 

CRT control/video window control circuitry 202 includes 
the CRT controller 220, window arbiter 221, and video 
display window controls 222. CRT controller 202 controls 
the refresh of the screen of display 106 and in particular the 
rastering of data from frame buffer 107 to display unit 107 
through DAC 210. In the preferred embodiment, CRT con- 
troller 220, through arbiter 218 and memory interface 219, 
maintains a constant stream of graphics data into graphics 
backend pipeline 205 from memory; video or playback 
graphics data is rastered oui only when a window has been 
reached by the display raster as determined by display 
position controls of window controls 222 (sec FIGS, 3 and 
5 and accompanying text) and CRT controller 220. As will 
be discussed in further detail below, the display of windows 
within the display according to the principles of the present 
invention is controlled in part by circuitry 202. 
Video [back-end] backend 

pipeline 204 receives a window of graph- 
ics video data defining a display window from the on-screen 
or off-screen spaces in firame buffer 107 through a pair of 
fint-in/fint-out memories 223 and 217 (in embodiments 
where buffer 217 is acdng as FIFO B). In the preferred 
embodiment, each FIFO receives the data for every other 
display line of data being generated for display on the 
display screen. For example, for a pair of adjacent lines n-1 
and n+1 in memory (although not necessarily adjacent on 
the display) for the display window, FIFO 223 receives the 
data defining window display line n-1 while FIFO 224 
receives the data defining window display line n+l. When 
buffer 217 is used as FIFO B, writes through video front end 
pipeline 200 are made through write buffer I 20f7 and 
multiplexer 235. Alternatively, if buffer 217 is used as write 
buffer n, then FIFO B is not imolementcd and cmly a single 
stream is processed by video [back-end] backend 

pipeline 204 (no Y 

mterpoladon is performed and Y expansion is by repUca- 
tion). As will be discussed further below (assuming both 
FIFO A and FIFO B are being used), one or more display 
lines, which falls between line n-1 and line n+1, may be 
selectively generated fay interpolation. Decoder circuitry 225 
receives two 32-bil packed words (as encoded by encoder 
214), one from each adjacent scan line in memory, &om 
FIFOs 223 and 217. Each 32-bit word, which represents four 
YCrCb pixels, is expanded and error diffused by decoder 
225 into four 16-bit YCrCb pixels. In modes where video 
60 data is stored in the frame buffer in standard 555 RGB or 16 
YCrCb data formats, decoder block 225 is bypassed. 

[Back-end] Backend 

video pipeline 204 further mcludes a Y mter- 
polator 226 and X interpolator 227. In the preferred embodi- 
ment, during Y zooming (expansion) Y interpolator 226 
65 accepts two vertically adjacent 1 6-bit RGB or YCrCb pixels 
from the decoder 225 and calculates one or more resampled 
output pixels using a four subpixel granularity. X interpo- 
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iator 227 during X zooming (expansion) accepts horizon- 
cally adjacent pixels from the Y interpolator 226 and calcu- 
lates one or more rcsamplcd output pixels using a four 
subpixcl granuJaniy, For data expansion using line replica- 
tion, Y inicqjolator 226 is bypassed. Y interpolator 226 and 5 
X interpolator 227 allow for the resizing of a video display 
window being generated from one to four times. 

The output of X interpolator 227 is passed to a color 
converter 228 which converts the YCrCb data into RGB data 
for delivery to output multiplexer 304. To reiterate, if 
graphics data is passed through- video pipeline .convener 228 
IS not used. 

[Back-end] Backend 

video circuitry 204 further includes pipeline 
control circuitry 229, overlay control circuitry 230 and 
output multiplexer 231, Pipeline control circuitry 239 con- 
trols the reading of data from video FIFOs 223 and 217, 
controls the generation of interpolation coefficients for use 
by X and Y interpolators 226 and 227 to resize the video 
window being pipelined, and times the transfer of data 
through the pipeline. Overlay control circuitry 230 along 
with control circuitry 202, controls the output of data ^ 
through output muitiplexer 231, including the overlay of the 
video window over the eraphics data output through the 
graphics [back-end] backend 

pipeline 205. A pixel doubler is provided 
to double the number of pixels being generated such that a 
1 280x1024 display can be driven. 25 
Graphics [back-endj backend 

pipeline 20S includes a first-in/first- 
out memory 232, atuibute controller 233, and color look-up 
tabic 234, Each 32-bil word output firom graphics FIFO 232 
is serialized into either 8-bit, 16-bit or 24-bit words. The 
8-bii words, typically composed of an ASCH code and an 30 
attribute code, arc sent to attribute controiler 233. When 
1 6-bit and 24-bit words, which arc typically color data, are 
serialized, those words are sem directly -overlay controls 
230. Attribute controller 233 performs such tasks as blinking 
and underlining operations in text modes. The eight bits 35 
output from attribute controller 233 are pseudo-color pixels 
used to index GLUT 234. GLUT 234 preferably outputs 
24-bit words of pixel data to output muldplexer 231 with 
each index. When video data is being pipelined through 
graphics backend pipeline 205 from the on-screen memory, 40 
GLUT 234 is bypassed. 
The eight bit pseudo-color pixels [are] output from attribute 
controller 233 are also sent to overly controls 230. In the 
preferred embodiment, data is continuously pipelined from 
on-screen memory through graphics [back-end] backend 

pipeline 205 45 
to the inputs of output multiplexer 231. Window data from 
off-screen memory however is only retrieved from memory 
and pipelined through video backend pipeline 204 when a 
window is being displayed. In other words, when a window 
has been reached, as determined by control bits set by CPU 50 
101 in VW control re^stcrs 222, video window display 
controls 222 generate addresses to retrieve the correspond- 
ing data from the oS'-screen memory space of frame buffer 
107. Proferabiy, video FIFOs 223 and 224 ate filled befm 
the raster scan actually reaches the display window such that 55 
the inidai' pixel data is available immediately once the 
window has been reached In order to insure that graphics 
memory data continues to be provided to gr^hics Iback-end] backend 
pipeline 205. video window display controls 222 "steal" 
page cycles between page accesses to the graphics memory. 60 
It should be noted that once the window has been reached 
the frequency of cycles used to retrieve window data 
increases over the number used to fill the video FIFOs when 
outside a window. When the frequency of window page 
accesses increases, video window display controls 222/ 65 
arbiter 221 preferably "steal" cycles from page cycles being 
used to write data into the frame buffer. 
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FIG, 3 is a more detailed functional block diagram 
emphasizing the circuitry controlling the overlay of data 
from graphics pipeline 205 with wmdow data from video 
pipeline 204. As discussed briefly above, the incuts to output 
5 multiplexer 231 are data from video [back-end] backend 

pipeline 204 

(pixel doubler 237), 16 or 24-bit color data directly from 
graphics back-end pipeline 205 seriaiizer 236 and 24-bit 
color data from the color look-up table 234. The output of 
data to DAC 210 through output multiplexer 231 is con- 

10 trolled by a latch 301 clocked by the video clock (VCLK). 
The remaining circuitry shown in FIG. 3, which will be 
discussed in further detail below, provide the necessary 
control signals to the control inputs of output multiplexer 
231 to select between the video and graphics pipelines. 

15 The graphics pseudo-pixels output from attribute control- 
ler 233 and the 16-bit or 24-bit graphics or video data output 
directly from seriaiizer 236 are provided to the inputs of 
color comparison circuitry 302, Also input to color com- 
parison [circuit] circuitry 

302 are 16 or 24-bil overlay color key bits 

20 stored in overlay color key register 303. Overiay color key 
register 303 resides within the address space of, and is 
loaded by, CPU 101. Depending on the mode, color com- 
parison circuitry 302 compares selected bits from the over- 
lay color key register 303 with either the 8 bits indexing 

25 look-up table 234 in the color look-up table mode (pseudo- 
color mode) or the 16-bits (24-bits in the alternate embodi- 
ment) passed directly from seriaiizer 236. It should be noted 
that in the illustrated embodiment, overlay color key register 
303 holds [24-bits of] 24 

of overlay color key bits, ei^t each for 

30 red, green, and [blue/index] blue index 

comparisons. The specific overlay 
color key bits compared with the input graphics data are 
provided in Table I: 
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Gieea<5:0> 


Bluc<4:0> 


3:8:g 






Blue<7:0> 



As shown in HG. 4A, a first embodiment of color 
comparison circuitry 303 performs the comparisons set forth 
in Table I as a set of XNOR opcraiions in scries with an AND 
45 operation. FIG. 4A depicts first comparison circuitry 400 for 
comparing the 8-bits of graphics pixels received in the 
look-up table mode from attribute comroller 233 with the 
8-bit blue/index overlay key bits being held in overiay key 
register 303. Second comparison drcmtry 401, perfoims the 
5Q required comparisons of Table I for the 16-bit data or 24-Mt 
received from seriaiizer 236, in, either a 5:5:5, 5:6:5, or 8:8:8 
format An overlay register 402 inchides a bit loaded by 
CPU 101 which is used by a selector 403, depending on the 
mode, to select for output, either the result of the compari- 
55 sons being made by comparison circuitry 400 in the color 
look-up table mode or the results of the comparisons being 
made by comparison dicuitry 401. In the illustrated embodi- 
ment, color conc5)arison drcoitry 303 processes data on a 
pixcl-by-pbccl basis and is rcsynchronized with both the 
60 gr^hics [back-end] backend 

pipeline 205 and the video back-end 
pipeUne 204 by having its outputs latched to the video clock 
(VCLK) by latches 404. 

The output of color comparison circuitry 303 is passed to 
the "K*' control input of overlay coiurol muldplexer 304. 
65 The "P" control input to muldplexer 304 is provided &om 
pixel position comparison circuitry 305. The data inputs to 
multiplexer 304 are coupled to an 8-bit overlay OP Code 
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(OOC) register 306, The ouipui of multiplexer 304 is used 
as one control input; to output multiplexer 304, which along 
wiih a single bit set by CPU 101 into output control register 
307, selects which of the data received at the data inputs of 
xultiplexer 231 will be output to DAC 210. 5 

Pixel position comparison circuitry 305 includes three 
inputs coupled respectively to video window 1 position 
control circuiL7 308, CRT position control circuiU7 309 and 
video wmdow 2 position control arcuitry 310. In the illus- 
trated embodiment, CRT position controller 309 is located iq 
wiihm CRT controller 220 while video window 1 position 
control circuitry and video window 2 position control cir- 
cuitry 310 are located within video display window controls 
222 (FIG. 2y CRT oosition control circuitry 309 includes 

counters r„ - . 1.1 . . 

fwhich,] which 

trade the position of the current pixel being 
generated for display. In the preferred embodiment, CRT 
position control circuitry 309 includes at least an x-pqsition 
counter which tracks the generation of each pixel along a 
given display line and a y-position counter which tracks the 
generation of each display line in a screen. The x-position 
counter may for example count pixels by counting each 
VCLK period between horizontal synchronization signal 
(HSYNC) controlling display unit 106. The y-position 
counter may for example count each HSYNC signi occur- 
ring between each vertical synchronization signal (VSYNC) 25 
contruiling the screen geoeradon on display unit 106. FIG. 
4B is an altercate embodiment of the color comparison 
circuitry of FIG. 3. In a first mode, 8 bits [are] 

from attribute 

controller 233 are passed through multiplexer to comparator 

406. Comparator 406 compares the received eight bits with ^ 
an 8-bi£ color key ia color key register 408; when the 
received 8-bils equal the 8-bit key 1, the output of compara- 
tor406go es active (high). In the first mode, control signal 
16BITGR is high (and the output of NOR gate 409 is 
consequently high) and an active output ftom comparator , 
406 is gated through AND gate 410. The output of AND gate 

410 is pzsscd to AND gate 411 and gated with the output 
from the pixel comparison circuitry 305. The output of AND 
gate 411 goes directly to the "B" control input of selector 
231 (in this embodiment nmltiplexer 304 and te^ster 306 ^ 
are eliminated). Thus, when the 8-bit graphics pixels output 
from attribute controller 233 of graphics backend 205 
matches the 8-bil color key 1 and the window has been 
reached as dctcnnined by pixel comparison circuitry 305, 
the pixel data output from video backend 204 are passed 
through selector 231. 

In a second mode, 16 bits are received from seriaHzer 236. 
The eight LSBs are passed through multiplexer 405 to 
comparat<»' 406 and the g^t MSBs passed to comparator 

407. Control signal 16BITNG is set high. When the LSBs » 
equal key 1 in coioriegister key 408 and the 8 MSBs equal 
key 2 in color key register 408, the outputs from compara- 
tors 406 and 407 are active (high). The ou^ut of AND gate 

411 then goes high when the output from pixel comparison 
circuitry 305, which is coupled to the "B" control input of 55 
selector 231, goes high. Thus, when the 16-bit pixel data 
output from seriaiizer 236 of graphics backend 205 tngtrh^ 
the 1 6-bit color key (keys 1 and 2) and a window has been 
reached, the output pixel data from video backend 204 are 
passed through selector 231. ^ 

FIG. 5 is an expanded functional block diagram of the 
video window position control circuits 308 and a corre- 
sponding portion of the gating of pixel position compare 
circuitry 305. Each position control circuit 3101^12 is 
coupled to a screen position x-register 500 and a screen 65 
position y-registcr 501, and includes a screen x-position 
couQter 502^ and a screen y-position counter 503. In the 
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preferred embodiment, registers 500 and 501 arc locaied 
within video window control registers 203. For the window 
corresponding to the given video window control circuitry 
308 or 310, registers 500 and 501 are loaded with a value 

^ representing the x and y screen position of the pixel in the 
upper left comer of that window (the starting pixel). Screen 
X- register 500 and screen y-register 501 in the preferred 
embodiment are loaded by CPU 101. The screen x-position 

10 counter 502 counts down from the value held in screen 
x-register 500 with each video clock when P is high for each 
display line and resets with each display horizontal synchro- 
nization signal (HS YNC) (Note thac when P is high the CRT 
count matches the position count). Screen y-position counter 
503 counts down from the value set into screen y-register 
501 for each horizontal sync signal (HSYNC) at the start of 
each display line and resets with each VS YNC at the stan of 
each new screen (The position counters are allowed to count 

2Q only when they match their perspective CRT), The counts 
values in the counters of CRT position control circuitry 309 
are compared pixel by pixel with the counts in screen 
x-position counter 502 and screen y-position 503 of each 
video window positioa control circuitiy 308 and 310. When 

25 both the x and y counts in the counters of CRT position 
control circuitry 309 match the corresponding x and y counts 
in respective counters 502 and 503 of cither video window 
control circuitry 308 or 310, the control signal P to muiti- 

^ plexer 304 is activated The activation of comrol signal P 
indicates that the raster scan on display 106 has reached the 
position of a pixel within the [window] window, 

, and data from video 
pipeline 205 may be painted depending on the value bdng 
held in overlay OF Code (OOQ register 306 and the K 

35 control ininits to multiplexer 304. 

A 4-bit OP Code loaded by CPU 101 into overlay OP 
Code register 306 in conjunction with the control signals 
applied to the "P" and "iC* control inputs to multiplexer 304 
control the presentation of an active (assumed high in the 
illustrated embodiment) control signal to dae "B" control 
input to output muitiplexw' 231. The other ("A") input to 
output multiplexer 231 receives a fait &om overlay mode 
register 402 (FIG, 4), as loaded by CPU 101. In the 

45 illustrated embodiment, the selection between the streams 
from the gr^hics and video backends at the 0,1,2 inputs to 
output multiplexer 304 in response to the signals presented 
at the corresponding control mputs "A" and "B** is in 
accordance with Table 0: 
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The OP Codes used in the illustrated embodiment, the 
65 effective overlay and the corresponding inputs to the control 
inputs of multiplexer 304 are listed in Table m (active state 
is assumedtl): 



13 



Overlay 0? Code MuUiplcxcr 307 
Register 309 Vduc Comroi Inputs 
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In the illustrated embodiment, if a Oh is written into OOC 
register 306 by CPU 101. only pixels from graphics pipeline 
205 arc pipelined through muldplexer 304. In this case any 
signals applied to the P and K control inputs to multiplexer 
304 have no effect (i.c„ will not result in a high output from 
multiplexer 304). In the illustrated embodiment, if an Ah is ^ 
written into OOC register 306, pixels from video pipeline 

204 will be passed to DAC 210 only when pixel position 
comparison circuitry 305 determines that the raster scan has 
reached a pixel in the window and hence the control signal 
going to the P input of muldplexer 304 has been acdvated 30 
If on the other hand, an 8h is written into OOC register 306, 
data is passed through output multiplexer 231 to DAC 210 
when pixel position comparison circuitry 305 detemunes 
that the raster scan has reached a pixel on the display screen 
within the window and color compare circuitry 302 has 35 
determined that the incoming data from graphics pipeline 

205 matches the overlay color key held in overiay color key 
register 303. In this case, the data from video pipeline 204 
is passed to DAC 210 when both the P and the K inputs to 
muldplexer 304 arc active. Fmally, when an OpCode of C is 40 
programmed into OOC register 306^ dau from video pipe- 
line 204 is passed if the incoming data from graphics 
pipeline 205 matches the overiay color key held in overlay 
color key register 303. In this case, the activation of ti» K 
control input activate the output of multiplexer 304 to switch 45 
the input of multiplexer 231 to pass die corresponding video 
pixels* 

Display control circuits embodying tijc principles of the 
present invention have substantial advantages over die prior 
art In particular, output control circuits built in accordance 50 
with die principles of die present invention allow for the 
flexible display of both graphics and video on die same 
screen. In particular, pixel position coix^)arison circuitry 305 
along witii video window position control circuits 308 and 
310 and CRT position control circuitry 309 allow for [a J 

one 55 
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Aithough the present invention and its advantages have 
been described in detail, it should be understood that various 
changes, substitutions and alterations [an] can 

be made herein 

without departing from the spirit and scope of the invention 
as defined by the appended claims. 



COLUMN 14 CONTINUED 
What is claimed is: 

1. A single Integra ted. "graphics and video conironer* 
adapted for driving a display t^^quentiaUy 

-comprising: 

an interface for receiving wards of pixel data, each said 
word associated with an address buffer; 
10 circuitry for writing each said word of said pixel data 
received by said interface to a one of on-screen and 
off-screen memory areas of a frame buffer; 

circuitry for selectively retrieving' 

as data is provided for display, 

said words from said 

on-screen and off- screen areas; 

15 a first pipeline for substantially continuously 

processing words of graphics data 
retrieved from said frame bufferi and 
a second pipeline for processing words of video data 
retrieved from said frame bufferso that the 

video data is ready for display once a display raster 
scan reaches a display position o£ a window. 

2. The controller of claim 1 and further comprising output 
selection circuitry for selecting for output between graphics 
data received from said first pipeline and data rweived from 
said second pipeline, said selection circuitry operable to: 
in a fint mode, pass data from said 

[first pipeline:] first pipeline; and 
^ in a second mode, pass data from said second pipeline 
when said data corresponds to a selected display posi- 
tion of a display window. 

3. The controller of claim 2 wherein said selection cir- 
cuitry is further operable to: 

30 in a third mode, pass data from said second pipeline when 
said data corresponds to said selected display position 
of said display window and data from said first pipeline 

' ^ match a color key. 

4. The controller of claim 3 wherein said selection cir- 
35 cuitry is further operable in a fourth mode to pass data from 

said second pipeline when data from said first pipeline 
match a color key, 

5. The controller of claim 1 wherein said circuitry for 
retrieving maintains a stream of graphics data to said first 

40 pipeline and provides video data to said second pipeline 
when a display raster scan reaches [said display] a display 
[said window] a window. position of 

6. The controller of claim 1 and further comprising: 
a video port for receiving real-time video A^ta; and 
drcuitry for generating an address to said memory at 

which said real-time video data is to be stored 

7. The controller of claim 1 whcrdn said second pipeline 
includes a first fiist-in-fint-ouC memory for receiving data 
for a fint display line of pixels in memory and a second 
first-in-first-out memory for receiving data from a second 
display line of [pixels memory] pixels in memory, 

8, The controller of claim 7 wherein said first display 
[line adjacent] line is adjacent 

in memory to said second display line, 

9, The controller of claim 7 

further comprising output selection circuitry 

wherein said output selection 

circtiitry comprises: 
an output selector for selecting between data from said 
second pipeline and data from said fint pipchne m 
response to a selection control signal; 

a register for maintaining a plurality of overlay control 
bits; 

window position control circuitry for selectively gener- 
ating a position control signal when a word of said data 
stream from said second pipeline falls within a display 
window; 

color comparison circuitry for comparing words of said 
data stream from said first pipeUne with a color key and 
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for providing in response a color comparison control 
signal; and 

a conirol selector for selectively providing [a 1 said 

selection 

control signal in response to said overlay control bits in 
said register and at least one of said position control and 5 
color comparison control signals. 

10. The controller of claim 9 wherein said window 
position control circuitry comprises: 

window position counters operable to increment from 
initial count values corresponding to a starting pixel of lO 
a display window as data representing each pixel in a 
display screen is pipelined through said overlay control 
circuitry; 

screen position counters operable to count as data repre- 
senung each pixel in said display screen is pipelined 15 
through said overlay control circuitry; and 

comparison circuiuy operable to compare a current count 
in said window position counters and a cuaent count in 
said screen position counters and selectively generate 
said position control signal in response. 20 

11. The controller of claim 9 wherein said color compari- 
son circuitry comprises: 

a color key register for storing bits composing said color 
key; and 

a plurality of AND-gates for comparing said words of said 25 
graphics data slrcd/n v.dth bits of said color key. 

12. The controller of claim 1 wherein said interface 
includes a dual-aperture port 

13. A single integrated controller comprising: 
cilfCuitry for writing selectively, on a word by word basis, 

each word of received 

data into [s] a _ 

selected one of on-screen and off-screen 
memory spaces of a frame buffer; 
a first port for receiving video and graphics data, a word 

of said data received with an address of said memory 3^ 

spaces directing said word to be processed as a word of 

video data or a word of graphics data; 
a second port for receiving real-time video data; 
circuitry for generating an address assodaied with a 

selected one of said memory spaces for a word of said 40 

real-time video data; 
circuitry for seiecdvcly retrieving said words of data 
on a word by word basis .from 

said on-screen and of -screen memoxy spaces as data is 

rastered for driving a diSplay:in a sequential fashion; 

a graphics backcnd pipeline for processing ones of said 
words of data representing graphics data retrieved from 
said frame buffer, 

a video backead pipeline , separate 

from said graphics backend pipeline, 

. for processing other ones of said 
words of data representing video data retrieved from ^ 
said frame buffer, said circuitry for retrieving always 
rastering a stream of data from said frame buffer to said 
graphics backend pipeline and rastering video data to 
said video backend pipeline ' [when] so that the 

video data is ready for display once 

a display raster scan 
reaches a display position of a window; and 
output selected circuitry for selecting for output between 
words of data ou^ut fom said graphics backend pipe- 
line and words of data output &om said video backend 

14. ^TtecontroUer of claim 13 wherein said output seiec- 60 
tor circuitry 

is further operable to select between graphics data output 
from a color look-up table and true color data output from 
said graphics pipeline. 

15, The controller of claim 13 wherein said output selec- 

toJ" circuitry 
is operable to; 

in a first mode, pass only a word of data output from said 
graphics pipeline; 
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in a second mode, pass a word of data output frona said 
video pipeline when said display raster scan has 
reached a display position corresponding to a window 
and a word of data from said graphics pipeline when 
5 said display raster scan is in any other display position; 
in a third mode, pass a word of data output from said 
video pipeline when said display raster scan has 
reached a display position corresponding to a window 
and a corresponding word of data from said graphics 
iQ pipeline matches a color key and a word of data from 
said graphics pipeline when said display raster scan is 
in any other display position; and 
in a fourth mode, pass a word of data from said video 
pipeline when said corresponding word of data from 
J 5 said graphics pipeline matches a color key and a word 
of data from said graphics pipeline when said [display 
raster scan is in any other display position] corresponding 
word does not match said color key. 

16. The controller of claim 13 wherein said video pipeline 
includes a first fint-in-first-out memory for receiving a 
plurality of words of data for a first display line of pixels in 
memory and a second first-in-flrst-out memory ' 

[or] for receiving 

a plurality of words of data Ifrom) for 

a second display line of 

pixels in memory. 

17. The controller of claim 16 wherein said first display 
line is stored adjacent tn memory to said second display line. 

^ 18. The controller of claim 13 wherein said output selec- 
tor circuitry comprises: 
a control selector having a plurality of control inputs 
coupled to a register, said register storing a plurality of 
overlay control bits; 
window position control circuitry coupled to a first con- 
trol input of said control selector, said window position 
control circuitry operable to selectively provide a first 
control signal to said first control input when a word of 
35 data being pipelined through said video pipeline falls 
within a display window; 
color comparison drcuitry operable to compare a word of 
data being pipelined through said graphics pipeline 
with a color key and provide in response a second 
40 control signal to a second control input of said control 
selector; and 

wherein said control selector is operable to provide an 
output selection control signal in response to at least 
one of said fint and second control signals and said 
45 overlay control bits being stored in said register. 

19. The circuitry of claim 18 wherein said ouQjut selector 
circuitry further includes a third coiUrol input coupled to 
certain bits of said graphics pipeline, said output selector 
circuitry further 

operable to select between data on said respective 
50 video and graphics pipelines in response to said certain bits 
presented to said selector circuitry. 

20. The circuitry of claim 18 wherein said window 
position control circuitry comprises: 

a window x-position counter operable to count from a 
loaded x-position value in response to a video clock, 
said x-position counter reloading in response to a display 
horizontal synchronization signal; 

a wujdow y-position counter operable to count from a 
loaded y-position value in response to said horizontal 
^ synchronization signal, said y-position counter reload- 
ing in response to a display vertical synchronization 
signal; . . 

CRT position circuitry operable tO maintam 

counts corre- 

65 Spending to a current display pixel; and 

comparison circuitry operable to compare current counts 
in said window counters with said current counts held 
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in said CRT position circuitry and generate in response 
said first control signal. 

21. The circuitry of claim 20 wherein said window 
posuion control circuitry further comprises an x-position 
register for holding said x-posiiion value for loading into 5 
said x-position counter and a y-position register for holding 
said y -position value for loading into said y-position 
counter. 

22. The circuitry of claim 13 wherein said color compari- 
son circuitry comprises: 

a color key register for storing a plurality of color key bits; 
and 

a plurality of XNOR-gaies each having at least one input 
coupled to a fait position in said color key register and 
at least one input coupled to said graphics data path, 

23. The circuitry of claim 13 wherein said video pipeline 
comprises: 

a first-in/first-out memory for receiving a fint stream of 

words of data from said frame buffer; 
a second first-in/f!rst-out memory disposed in parallel 20 

with said first first-in/first-out memory for receiving a 

second stream of words of data from said frame buffer, 

and 

interpolation circuitry for selectively generating an addi- 
tional word of data by interpolating a word of said first 25 
stream and a word of Said SCCOnd 

stream data output from 
said first and second first-in/first-out memories. 

24. The controller of claim 13 wherein said first [pore] port 
comprises a dual-aperture port 

25. A display system comprising: 30 
a first backend pipeline for processing data; 

a second backend pipeline for processing graphics data 
disposed in parallel to said first processing pipeline; 

a multi-format &ame buffer memory having on-screen 
and off-scieen areas each operable to 

to allow said frame buffer to - - 

simultaneously 
store data in graphics and video fomats; 

[a] an 

input port for receiving both graphics and video data, 
each word of said data associated with an address 
directing said word to be processed as either graphics ^ 
or video data; 

circuitry for writing a word of said playback data into a 
selected one of said areas of said multi-format memory; 

memory control circuitry for controlling the transfer of 
data between said first backend pipeline and said frame 45 
buffer and between said second backend pipeline and 
said frame buffer; 

a display unit; and 

overiay control ciicuilry for selecting for output to said 
display unit between data provided by said first back- 
end pipeline and data provided by said second backend 
pipeline, 

26. The display system of claim 25 wherein said second 
backend pipeline includes: 

a first first-in-first-out memory for receiving first selected 
data; 

a second first-in-first-out memory disposed in parallel to 
said first first-in-first-out memory for receiving second 
selected j ^ ta* and 60 

interpolation data for generating additional data by inter- 
polating data ouQHit from said respective first and 
second first-in-first-otit memories. 

27. The display system of claim 26 wherein said second 
backend pipeline further comprises color conversion cir- 65 
cuitry for converting data received from said frame buffer in 

a video format to a graphics format. 
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28, The display system of claim 25 [and] 

further comprising 

a video front-end pipeline for inputting video data into a 
selected one of on-screen and off-screen spaces of said frame 
buffer comprising: 
5 a video data port for receiving video data from a real time 

data sotiTce; and 

input control circuitry for receiving framing signals asso- 
ciated with said real time data and generating corre- 
sponding addresses to said selected one of said spaces 
10 in response, 

29, The display system of claim 28 wherein said video 
front-end pipeline ftxrther comprises encoding circuitry for 
packing said video data prior to storage in said selected one. 

of said 

spaces. 

J5 30, The display system of claim 28 wherein said video 
front-end pipeline further [comprising] comprises 
^ , . ^ > multiplexing circuitry 

for selectmg between video data received through said video 
data port and data received &om [said] a dual 

3L The display system of claim 30 whJKd^Seo 
?onveS^"h'^^^ 

converting graphics data received through said duaI-a™iiT«. 
port ^o a video fonnat for storage in sail^SS^l^K 

hJ^n?* r^^^^ ^y'^''"' 25 wherein said first 

backcad pipeline processes playback video. 

rnmnJ^*" !S '^'^"^ ^ s^id input port 

compnses a dual-aperture intmt port ^ 

34. A single integrated 

display data processing system comprising: 
circuitry for writing data into an on-screen space of a 
frame buffer; 

^° circuitry for writing data into an off-sciccn space of said 
frame buffer 

a video pipeline for processing data output from a selected 
one of said on-screen and off-scxeen spaces comprising: 
a first first-in-first-out memory for receiving selected 

data from said selected space; 
a second fint-in-first-out memory disposed in parallel 
to said fint first-in-first-out memory for receiving 
other selected data from said selected spac<^ and 
an interpolator for generating additional data by inter- 
40 polating data output from said respective first and 

second first-in-first-out memories; 
a grapfaici pipeline disposed in parallel to 

and separate from 

- . .said video 

pipeline for processing data ou^ from a selected one 
of said on-screen and off-screen spaces; and 
an ou^mt selector for selecting between data ou^ut from 
said video pipeline and data output from said gr^faics 
pipeline. 
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COLUMN 18 CONTINUED 



35. The system of claim 34 [and] 

further comprising 

selection control circuitry for generating an output control 
signal for controlling said output selector comprising: 
a control selector having a plurality of data inputs coupled 
to a register, said register for storing a plurality of 
overlay control bits; and 
55 color comparison circuitry operable to compare bits of 
data output &om said graphics pipeline with a color key 
and provide in response a control signal to a control 
input of said control selector. 

36. The system of claim 34 [andj f^,^ comprising 
go window posidon control circuitry operable to provide a 

second control signal to a second control -input of said 
control selector when data &om said video pipeline falls 
within a displ^ window. 

37. A' single integrated 

/display controller comprising: 
65 circuitry for selecdvcly retrieving data fom an associated 

multi-format frame buffer , the frame buffer having 
separate storage locations respectively operable for allowing 

^ 'for simultaneously 

[snoring] storing 
graphics and video data; 

liir said frame buffer; 
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a first pipeline for processing words of graphics data 
solectivcly retrieved from said frame buffer; and 

a second pipeline ^ sep^j^t^ f^om the first pipeline, 

for processing words of video data 
sciecuvciy retrieved from said frame buifer. 

38. The controller of claim 37 wticrcin said first and ^ 
second pipelines are disposed in parallel and concurrently 
process data. 

39. The controller of claim 38 [and] 

further comprising 

output selection circuitry for selecting for output between 
graphics data received from said first pipeline and video data 
received from said second pipeline. 

40. The controller of claim 37 wherein said frame buffer 
is partitioned into on-screen and o£f-screen areas, each of 
said on-screen and off-screen areas operable to 

/ allow the buffer to 

simulta- 
neously store both graphics and video data, 

41 The controller of claim 37 wherein said circuitry for 
seiecuveiy retrieving is operable to retrieve a substantially 

constant 

stream of graphics data from said frame buffer and provide 
said stream of graphics data to said first pipeline. 

42 The controller of claim 41 wherein said circuitry for 
selectively retrieving is operable to retrieve at least one said 
word of video data from said frame buffer and provide said 
at least one word of said video data to said second pipeline, 
only when said display controller is generating a video ^ 
display window, 

43 A display controller for interfacing a mulli-fonnat 
frame buffer and a display device, the multi-format frame 

buffer having on-screen and off-screen areas cacb^perabie for aiiowng 

simultaneously storing both graphics and video pixel data in the frame bulTer^ 

said 

30 

display controller compnsing: 
circuitry for selectively retrieving pixel data from a 
selected one of said on-screen and off-screen areas oi 
said frame buffer; 
a graphics backend pipeline for processing graphics d^ 35 
retrieved from said selected one of said areas of said 
frame buffer; 
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a video backend pipeline for processing video data 
retrieved from said selecied one of said areas of said 
frame buffer, and 
^ an output selector for selectively passing to said display 
device data received from said graphics or video back- 
end pipelines. 

44. The display controller of claim 43 wherein said 
circuitry for selectively retrieving is operable to retrieve at 

10 least one said word of video data from said frame buffer and 
provide said at least one said word of video data [no] to 

said 

second pipeline only when said display controller is gener- 
ating a video display window. 

45. The display controller of claim 43 wherein said output 
selector is operable to: 

in a first [modes] mode 

pass data from said graphics pipeline; and 
in a second [modes} mode 

pass data from said video pipeline 
when a display position corresponding to a video 
20 display window has been reached. 

46. The display window of claim 43 wherein said output 
selector is operable to: 

in a first mode, pass data &om said graphics pipeline; and 
25 in a second mode, pass data from said video pipeline 
when a display position corresponding to a video 
display window has been reached and data from said 
graphics pipeline match a color key. 

47. The display controller of claim 43 wherein said output 
30 selector is operable to; 

in a first mode, pass data from said graphics pipeline; and 
in a second mode, pass data from said video pipeline 
when data from said graphics pipeline matches a color 
3j key. 

48. The controller of claim 42 in which graphics data is substantially 
continuously retrieved for a display while video data is retrieved onW for 
the video display window. 
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ABSTRACT 



A graphics and video controller 105 is provided which 
includes a dual aperture interface 206 for receiving words of 
graphics and video pixel data, each word of such data 
associated with an address directing that word to be pro- 
cessed as either graphics or video data. Circuitry 200. 201, 
202, 207, 208 is provided for writing a word of the pixel data 
received from the interface 206 to a one of the on- and 
off-screen memory areas conesponding to the address asso- 
ciated with the received word. Circuitry 201, 202 is provided 
for selectively retrieving gn^hics and video data from the 
on-screen and off-screen memory areas. A first pipeline 205 
is provided for jKocessing data received fit>m the on-screen 
area of frame buffer 107 while a second pipeline 204 is 
provided for processing data retrieved from the off-screen 
area of the frame buffer. 
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Assistant Commissioner for Patents 
Washington, DC 20231 

Sir: 

Cirrus Logic, Inc., Assignee of the entire interest in the above-identified patent, hereby 
declares: 

The post office address of Cirrus Logic, Inc. is stated below; and that 
Cirrus Logic, Inc. verily beUeves that inventors Robert M. Nally of Piano, Texas and John 
C. Schafer of Wylie, Texas are the original and first joint inventors of the invention entitled: 
APPARATUS, SYSTEMS AND METHODS FOR CONTROLLING GRAPHICS AND VIDEO 
DATA IN MULTIMEDIA DATA PROCESSING AND DISPLAY SYSTEMS, described and 
claimed in the U.S. Patent No. 5,598,525, filed January 23, 1995, and issued on January 28, 1997, 
that Cirrus Logic, Inc. understands the content of the specification filed therein, that Cirrus Logic, 



Inc. has reviewed and understands the content of the specification including the claims originally 
patented, as well as the claims referred to in this Declaration, that Cirrus Logic, Inc. does not know 
and does not believe the same was ever known or used in the United States of America before the 
invention thereof by said inventors, or patented or described in any printed pubhcation in any 
country before their invention thereof or more than one year prior to the filing date of the 
appUcation which matured into U.S. Patent No. 5,598,525; that the same was not in public use or on 
sale in the United States of America more than one year prior to said filing date to the best of Cirrus 
Logic, Inc.'s knowledge, information and belief, that the invention has not been patented or made 
the subject of an inventor's certificate issued before said filing date, in any country foreign to the 
United States of America on an application filed by Cirrus Logic, Inc. or Cirrus Logic, Inc.'s legal 
representatives or assigns more than twelve months prior to said filing date, that Cirrus Logic, Inc. 
acknowledges a duty to disclose information of which Cirrus Logic, Inc. is aware which is material 
to the examination of this application, and that no application for patent or inventor's certificate on 
this invention has been filed in any country foreign to the United States of America prior to the 
original application Serial No. 376,919, filed on January 23, 1995 by us or our legal representatives 
or assigns, except as follows: NONE 

Our original U.S. Patent 5,598,525, issued January 28, 1997, which matured from said 
Serial No. 376,919, is believed to be wholly or partly inoperative by reason of the patentees 
claiming the invention imprecisely and with aspects of the invention recited only implicitly through 
error and without any deceptive intention. 

That Applicants claims are wholly or partly inoperative by failure to assert claims of the 
following language: 
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1. A single integrated graphics and video controller adapted for driving a display sequentially 
comprising: 

an interface for receiving words of pixel data, each said word associated with an address buffer; 

circuitry for writing each said word of said pixel data received by said interface to a one of on- 
screen and off-screen memory areas of a frame buffer; 

circuitry for selectively retrieving, as data is provided for display, said words from said on-screen 
and off-screen areas; 

a first pipeline for substantially continuously processing words of graphics data retrieved from 
said frame buffer; and 

a second pipeline for processing words of video data retrieved from said frame buffer so that the 
video data is ready for display once a display raster scan reaches a display position of a window. 

2. The controller of claim 1 and further comprising output selection circuitry for selecting for 
output between graphics data received from said first pipeline and data received from said 
second pipeline, said selection circuitry operable to: 

in a first mode, pass data from said first pipeline: and 

in a second mode, pass data from said second pipeline when said data corresponds to a selected 
display position of a display window. 

3. The controller of claim 2 wherein said selection circuitry is further operable to: 

in a third mode, pass data from said second pipeline when said data corresponds to said selected 
display position of said display window and data from said first pipeline match a color key. 

4. The controller of claim 3 wherein said selection circuitry is further operable in a fourth mode 
to pass data from said second pipeline when data from said first pipeline match a color key. 

5. The controller of claim 1 wherein said circuitry for retrieving mauitains a stream of graphics 
data to said first pipeline and provides video data to said second pipeline when a display raster 
scan reaches said display position of said window. 

6. The controller of claim 1 and further comprising: 
a video port for receiving real-time video data; and 

circuitry for generating an address to said memory at which said real-time video data is to be 
stored. 

7. The controller of claim 1 wherein said second pipeline includes a first first-in-first-out 
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memory for receiving data for a first display line of pixels in memory and a second first-in-first- 
out memory for receiving data from a second display line of pixels memory, 

8. The controller of claim 7 wherein said first display line adjacent in memory to said second 
display line. 

9. The controller of claim 7 wherein said output selection circuitry comprises: 

an output selector for selecting between data from said second pipeline and data from said first 
pipeline in response to a selection control signal; 

a register for maintaining a plurality of overlay control bits; 

window position control circuitry for selectively generating a position control signal when a 
word of said data stream from said second pipeline falls within a display window; 

color comparison circuitry for comparing words of said data stream from said first pipeline with 
a color key and for providing in response a color comparison control signal; and 

a control selector for selectively providing a said selection control signal m response to said 
overlay control bits in said register and at least one of said position control and color comparison 
control signals. 

10. The controller of claim 9 wherein said window position control circuitry comprises: 

window position counters operable to increment from initial count values corresponding to a 
starting pixel of a display window as data representing each pixel in a display screen is pipehned 
through said overlay control circuitry; 

screen position counters operable to count as data representing each pixel in said display screen 
is pipelined through said overlay control circuitry; and 

comparison circuitry operable to compare a current count in said window position counters and a 
current count in said screen position counters and selectively generate said position control 
signal in response. 

1 1 . The controller of claim 9 wherein said color comparison circuitry comprises: 
a color key register for storing bits composing said color key; and 

a plurality of AND-gates for comparing said words of said graphics data stream with bits of said 
color key. 

12. The controller of claim 1 wherein said interface includes a dual-aperture port. 

13. A single integrated controller comprising: 
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circuitry for writing selectively, on a word by word basis, each word of received data into a 
selected one of on-screen and off-screen memory spaces of a frame buffer; 

a first port for receiving video and graphics data, a word of said data received with an address of 
said memory spaces directing said word to be processed as a word of video data or a word of 
graphics data; 

a second port for receiving real-time video data; 

circuitry for generating an address associated with a selected one of said memory spaces for a 
word of said real-time video data; 

circuitry for selectively retrieving said words of data on a word by word basis fi-om said on- 
screen and off-screen memory spaces as data is rastered for driving a display in a sequential 
fashion; 

a graphics backend pipeline for processing ones of said words of data representing graphics data 
retrieved from said frame buffer; 

a video backend pipeline, separate from said graphics backend pipeline, for processing other 
ones of said words of data representing video data retrieved from said frame buffer, said circuitry 
for retrieving always rastering a stream of data from said frame buffer to said graphics backend 
pipeline and rastering video data to said video backend pipeline so that the video data is ready 
for display once a display raster scan reaches a display position of a window; and 

output selector circuitry for selecting for output between words of data output from said graphics 
backend pipeline and words of data output from said video backend pipeline. 

14, The controller of claim 13 wherein said output selector is further operable to select between 
graphics data output from a color look-up table and true color data output from said graphics 
pipeline. 

15. The controller of claim 13 wherein said output selector is operable to: 

in a first mode, pass only a word of data output from said graphics pipeline; 

in a second mode, pass a word of data output from said video pipeline when said display raster 
scan has reached a display position corresponding to a window and a word of data from said 
graphics pipeline when said display raster scan is in any other display position; 

in a third mode, pass a word of data output from said video pipeline when said display raster 
scan has reached a display position corresponding to a window and a corresponding word of data 
from said graphics pipeline matches a color key and a word of data from said graphics pipeline 
when said display raster scan is in any other display position; and 
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in a fourth mode, pass a word of data from said video pipeline when said corresponding word of 
data from said graphics pipeline matches a color key and a word of data from said graphics 
pipehne when saidcorresponding word does not match said color key. 

16. The controller of claim 13 wherein said video pipeline includes a first first-in-first-out 
memory for receiving a plurality of words of data for a first display line of pixels in memory and 
a second first-in-first-out memory for receiving a plurality of words of data for a second display 
line of pixels in memory. 

17. The controller of claim 16 wherein said first display Hne is stored adjacent in memory to said 
second display line. 

18. The controller of claim 13 wherein said output selector circuitry comprises: 

a control selector having a plurality of control inputs coupled to a register, said register storing a 
plurality of overlay control bits; 

window position control circuitry coupled to a first control input of said control selector, said 
window position control circuitry operable to selectively provide a first control signal to said first 
control input when a word of data being pipelined through said video pipeline falls within a 
display window; 

color comparison circuitry operable to compare a word of data being pipelined through said 
graphics pipeline with a color key and provide in response a second control signal to a second 
control input of said control selector; and 

wherein said control selector is operable to provide an output selection control signal in response 
to at least one of said first and second control signals and said overlay control bits being stored in 
said register. 

19. The circuitry of claim 18 wherein said output selector circuitry further includes a third 
control input coupled to certain bits of said graphics pipeline, said output selector further 
operable to select between data on said respective video and graphics pipelines in response to 
said certain bits presented to said selector circuitry. 

20. The circuitry of claim 18 wherein said window position control circuitry comprises: 

a window x-position counter operable to count from a loaded x-position value in response to a 
video clock, said x-position counter reloading in response to display horizontal synchronization 
signal; 

a window y-position counter operable to count from a loaded y-position value in response to said 
horizontal synchronization signal, said y-position counter reloading in response to a display 
vertical synchronization signal; 

CRT position circuitry operable maintain counts corresponding to a current display pixel; and 
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comparison circuitry operable to compare current counts in said window counters with said 
current counts held in said CRT position circuitry and generate m response said first control 
signal. 

21. The circuitry of claim 20 wherein said window position control circuitry further comprises an 
x-position register for holding said x-position value for loading into said x-position counter and a 
y-position register for holding said y-position value for loading into said y-position counter. 

22. The circuitry of claim 13 wherein said color comparison circuitry comprises: 
a color key register for storing a plurality of color key bits; and 

a plurality of XNOR-gates each having at least one input coupled to a bit position in said color 
key register and at least one input coupled to said graphics data path. 

23. The circuitry of claim 13 wherein said video pipeline comprises: 

a first-in/first-out memory for receiving a first stream of words of data fi-om said firame buffer; 

a second first-in/first-out memory disposed in parallel with said first first-in/first-out memory for 
receiving a second stream of words of data from said frame buffer; and 

interpolation circuitry for selectively generating an additional word of data by interpolating a 
word of said fnst stream and a word of second stream data output from said first and second 
first-in/first-out memories. 

24. The controller of claim 13 wherein said first pore comprises a dual-aperture port. 

25. A display system comprising: 

a first backend pipeline for processing data; 

a second backend pipeline for processing graphics data disposed in parallel to said first 
processing pipeline; 

a multi-format frame buffer memory having on-screen and off-screen areas each operable to 
allow said frame buffer simultaneously store data in graphics and video formats; 

a input port for receiving both graphics and video data, each word of said data associated with an 
address directing said word to be processed as either graphics or video data; 

circuitry for writing a word of said playback data into a selected one of said areas of said multi- 
format memory; 

memory control circuitry for controlling the transfer of data between said first backend pipeline 
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and said frame buffer and between said second backend pipeline and said frame buffer; 
a display unit; and 

overlay control circuitry for selecting for output to said display unit between data provided by 
said first backend pipeline and data provided by said second backend pipeline. 

26. The display system of claim 25 wherein said second backend pipeline includes: 
a first first-in-first-out memory for receiving first selected data; 

a second first-in-fu-st-out memory disposed in parallel to said first first-in-first-out memory for 
receiving second selected data; 

interpolation data for generating additional data by interpolating data output from said respective 
first and second first-in-first-out memories. 

27. The display system of claim 26 wherein said second backend pipeline fiirther comprises 
color conversion circuitry for converting data received from said frame buffer in a video format 
to a graphics format. 

28. The display system of claim 25 and further comprising a video front-end pipeline for 
inputting video data into a selected one of on-screen and off-screen spaces of said frame buffer 
comprising: 

a video data port for receiving video data from a real time data source; 

input control circuitry for receiving framing signals associated with said real time data and 
generating corresponding addresses to said selected one of said spaces in response. 

29. The display system of claim 28 wherein said video front-end pipeline ftirther comprises 
encoding circuitry for packing said video data prior to storage in said selected one spaces. 

30. The display system of claim 28 wherein said video front-end pipeline further comprising 
multiplexing circuitry for selecting between video data received through said video data port and 
data received from said dual aperture port. 

31. The display system of claim 30 wherein said video front end pipeline further comprises 
conversion circuitry for converting graphics data received through said dual-aperture port to a 
video format for storage in said selected one of said spaces. 

32. The display system of claim 25 wherein said first backend pipeline processes playback video. 

33. The display system of claim 25 wherein said input port comprises a dual-aperture input port. 

34. A single integrated display data processing system comprising: 



8 



circuitry for writing data into an on-screen space of a frame buffer; 
circuitry for writing data into an off-screen space of said frame buffer; 

a video pipeline for processing data output from a selected one of said on-screen and off-screen 
spaces comprising: 

a first first-in-first-out memory for receiving selected data from said selected space; 

a second first-in-first-out memory disposed in parallel to said first first-in-first-out memory for 
receiving other selected data from said selected space; and 

an interpolator for generating additional data by interpolating data output from said respective 
first and second first-in-first-out memories; 

a graphics pipeline disposed in parallel to and separate from said video pipeline for processing 
data output from a selected one of said on-screen and off-screen spaces; and 

an output selector for selecting between data output from said video pipeline and data output 
from said graphics pipeline. 

35. The system of claim 34 and fixrther comprising selection control circuitry for generating an 
output control signal for controlUng said output selector comprising: 

a control selector having a plurality of data inputs coupled to a register, said register for storing a 
plurality of overlay control bits; and 

color comparison circuitry operable to compare bits of data output firom said graphics pipeline 
with a color key and provide in response a control signal to a control input of said control 
selector. 

36. The system of claim 34 and further comprising window position control circuitry operable to 
provide a second control signal to a second control input of said control selector when data from 
said video pipeline falls within a display window. 

37. A single integrated display controller comprising: 

circuitry for selectively retrieving data from an associated multi-format frame buffer, the frame 
buffer having separate storage locations respectively operable for allowing simultaneously 
storing graphics and video data in said frame buffer; 

a first pipeline for processhig words of graphics data selectively retrieved from said frame buffer; 
and 

a second pipeline, separate from the first pipeline, for processing words of video data selectively 
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retrieved from said frame buffer. 

38. The controller of claim 37 wherein said first and second pipelines are disposed in parallel and 
concurrently process data. 

39. The controller of claim 38 and fiirther comprising output selection circuitry for selecting for 
output between graphics data received from said first pipeline and video data received from said 
second pipeline. 

40. The controller of claim 37 wherein said frame buffer is partitioned into on-screen and off- 
screen areas, each of said on-screen and off-screen areas operable to allow the buffer to 
simultaneously store both graphics and video data. 

41. The controller of claim 37 wherein said circuitry for selectively retrieving is operable to 
retrieve a substantially constant stream of graphics data from said frame buffer and provide said 
stream of graphics data to said first pipeline. 

42. The controller of claim 41 wherein said circuitry for selectively retrieving is operable to 
retrieve at least one said word of video data from said frame buffer and provide said at least one 
word of said video data to said second pipeline, only when said display controller is generating a 
video display window. 

43. A display controller for interfacing a multi-format frame buffer and a display device, the 
multi-format frame buffer having on-screen and oflf-screen areas each operable for allowing 
simultaneously storing both graphics and video pixel data in the frame buffer, said display 
controller comprising: 

circuitry for selectively retrieving pixel data from a selected one of said on-screen and off-screen 
areas of said frame buffer; 

a graphics backend pipeline for processing graphics data retrieved from said selected one of said 
areas of said frame buffer; 

a video backend pipeline for processing video data retrieved from said selected one of said areas 
of said frame buffer; and 

an output selector for selectively passing to said display device data received from said graphics 
or video backend pipelines. 

44. The display controller of claim 43 wherein said circuitry for selectively retrieving is operable 
to retrieve at least one said word of video data from said frame buffer and provide said at least 
one said word of video data no said second pipeline only when said display controller is 
generating a video display window. 

45. The display controller of claim 43 wherein said output selector is operable to: 
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in a first modes pass data from said graphics pipeline; and 

in a second modes pass data from said video pipeline when a display position corresponding to a 
video display window has been reached. 

46. The display window of claim 43 wherein said output selector is operable to: 
in a first mode, pass data from said graphics pipeline; and 

in a second mode, pass data from said video pipeline when a display position corresponding to a 
video display window has been reached and data from said graphics pipeline match a color key. 

47. The display controller of claim 43 wherein said output selector is operable to: 
in a first mode, pass data from said graphics pipeline; and 

in a second mode, pass data from said video pipeline when data from said graphics pipeline 
matches a color key. 

48. The controller of claim 42 in which graphics data is substantially continuously retrieved for 
a display while video data is retrieved only for the video display window. 

That the failure to assert claims of this nature occurred without deceptive intention because 
Assignee, through arguments by opposing counsel in litigation and through an Initial Decision of an 
Administrative Law Judge at the Federal Trade Commission, became aware that the original 
patented claims could be misconstrued as indefinite or misconstrued to read on prior art. 

We hereby revoke all prior powers of attorney and appoint the following attorneys to 
prosecute this application and to transact all business in the Patent and Trademark Office connected 
therewith: 

David L. Stewart, Reg. No. 37,578; Gene Z. Rubinson, Reg. No. 33,351; Eugene J. 
Mohnelli, Reg. No. 42,901 of McDermott, Will & Emery; and 

J.P, Violette, Reg. No. 33,042; Steven A. Shaw, Reg. No. 39,368; Dan A. Shifrin, Reg. No. 
34,473 of Cirrus Logic, Inc. 
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All future correspondence connected therewith should be addressed to the following 
address: 

David L. Stewart, Esq. 
McDermott, Will & Emery 
600 13**^ Street, N. W. 
Washington, DC 20005-3096 

The undersigned hereby declare that all statements made herein of his own knowledge are 
true and that all statements made on information and beUef are beheved to be true; and fiirther that 
these statements and the like so made are punishable by fine or imprisonment, or both, under 
Section 1001 of Title 18 of the United States Code and that such willful false statements may 
jeopardize the vahdity of the application or any patent issued thereon. 

CIRRUS LOGIC, INC. 

By: 

Company Officer's Title: 

Date: 
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CERTIFICATE UNDER 37 CFR §3.73f 



Applicant: Robert M. Nally, et al. 

Application No.: 376,919 Filed: January 23, 1995 

For: APPARATUS, SYSTEMS AND METHODS FOR CONTROLLING GRAPfflCS AND 
VIDEO DATA IN MULTIMEDIA DATA PROCESSING AND DISPLAY SYSTEMS 

Cirrus Logic, Inc. a Corporation 

(Name of Assignee) (Type of Assignee) 



certifies that it is the assignee of the entire right, title and interest in the patent appUcation identified 
above by virtue of : 

A^ An assignment from the inventor(s) of the patent application identified above. The 
assignment was recorded in the Patent and Trademark Office at Reel 7326, Frame 0692^ 
That assignment is subject to a security interest in favor of the Bank of America National 
Trust and Savings as agent, dated August 29, 1996 and recorded at Reel 81 13, Frame OOOL 



OR 



B. □ A chain of title from the inventor(s), of the patent application identified above, to the current 
assignee as shown below: 



1. From: dls To: 

The document was recorded in the Patent and Trademark Office at Reel , Frame , or 
for which a copy thereof is attached. 

2. From: To: 

The document was recorded in the Patent and Trademark Office at Reel , Frame , or 
for which a copy thereof is attached. 

3. From: To: 

The document was recorded in the Patent and Trademark Office at Reel , Frame , or 
for which a copy thereof is attached. 

I I Additional documents in the chain of title are Usted on a supplemental sheet. 

O Copies of assignments or other documents in the chain of title are attached. 



The undersigned has caused a review to be made of all the documents in the chain of title of the 
patent apphcation identified above and, to the best of undersigned's knowledge and belief, title is in 
the name of the assignee identified above. 

The undersigned (whose title is suppUed below) is empowered to act on behalf of the assignee. 



I hereby declare that all statements made herein of my own knowledge are true, and that all 
statements made on information and belief are beheved to be true; and fiirther, that these statements 
are made with the knowledge that willful false statements, and the like so made, are punishable by 
fine or imprisonment, or both, under Section 1001, Title 18 of the United States Code, and that such 
willful false statements may jeopardize the validity of the apphcation or any patent issuing thereon. 

Date: 

Name: 

Title: 



Signature 
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PATENT 



m THE UNITED STATES PATENT AND TRADEMARK OFFICE 



In re Reissue of U.S. Patent 5,598,525 



Issued January 28, 1997 



To Robert M. Nally, et al. 



Based on Serial No. : 376,919 



Group Art Unit: 



Filed: January 23, 1995 



Examiner: 



For: APPARATUS, SYSTEMS AND METHODS FOR CONTROLLING GRAPHICS AND 
VIDEO DATA IN MULTIMEDIA DATA PROCESSING AND DISPLAY SYSTEMS 



Box 7 

Assistant Commissioner for Patents 
Washington, DC 20231 

Sir: 

In accordance with the provisions of 37 CFR 1 . 172, Cirrus Logic, Inc. Assignee of the entire 
interest of U.S. Patent 5,598,525 to Robert M. Nally et al., by virtue of an Assignment from the 
inventors to Cirrus Logic, Inc. dated January 23, 1995 and recorded at Reel 7326, Frame 0692, 
subject to a Security Agreement dated August 29, 1996 in favor of Bank of America National Trust 
and Savings as agent, recorded at Reel 81 13, Frame 0001, does hereby consent to the filing of the 
accompanying Reissue AppUcation. 



ASSIGNEE'S CONSENT TO REISSUE 



In witness whereof the undersigned hereby sets his hand and seal. 

Signature: 

CIRRUS LOGIC, INC. 

By: 

Title: 
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PATENT 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re Reissue of U.S. Patent 5,598,525 
Issued January 28, 1997 
To Robert M. Nally, et al. 
Based on Serial No. : 376,91 9 
Filed: January 23, 1995 



Group Art Unit: 
Examiner: 



For: APPARATUS, SYSTEMS AND METHODS FOR CONTROLLING GRAPfflCS AND 
VIDEO DATA IN MULTIMEDIA DATA PROCESSING AND DISPLAY SYSTEMS 



OFFER TO SURRENDER LETTERS PATENT 



Box 7 

Assistant Commissioner for Patents 
Washington, DC 20231 



Sir: 

In accordance with the provisions of 37 CFR 1.178, the undersigned Assignee of the 
accompanying reissue appUcation for the reissue of U.S. Letters Patent Number 5,598,525 for 
APPARATUS, SYSTEMS AND METHODS FOR CONTROLLING GRAPHICS AND VIDEO 
DATA IN MULTIMEDIA DATA PROCESSING AND DISPLAY SYSTEMS granted on January 



28, 1997, to Robert M. Nally, John C. Schafer and assigned to Cirrus Logic, Inc. of Fremont, 
California, by assignment of the entire interest, hereby offers to surrender said Letters Patent. 

CIRRUS LOGIC, INC. 

By: 

Title: 

Date: 
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PATENT 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



Reissue of U.S. Patent 5,598,525 
Issued to Robert M. NALLY, et al. 
Issue Date: January 28, 1997 
Based on Serial No.: 376,919 
Filed: January 23, 1995 




Group Art Unit: None 
Examiner: None 



For: APPARATUS, SYSTEMS AND METHODS FOR CONTROLLING GRAPfflCS AND 
VIDEO DATA IN MULTIMEDL^ DATA PROCESSING AND DISPLAY SYSTEMS 



Honorable Commissioner of 
Patents and Trademarks 
Washington, D. C. 20231 



LETTER REGARDING REISSUE APPLICATION 



Sir: 



letter. 



A Reissue Application based on U.S. Patent 5,598,525 to Nally et al. accompanies this 



Litigation in two cases is relevant to this reissue application. They are: 

IN THE MATTER OF CERTAIN VIDEO GRAPHICS 
DISPLAY CONTROLLERS AND PRODUCTS CONTAINING 
SAME, United States International Trade Commission 
Investigation No. 337-TA-412, and 

CIRRUS LOGIC, INC., VS. ATI TECHNOLOGIES, 
INC., United States District Court for the Northern District of 
California, San Francisco Division, Docket No. C-98-2700SI. 



The ITC litigation has concluded with an INITIAL DETERMINATION and a subsequent 
NOTICE OF COMMISSION DETERMINATION NOT TO REVIEW THE BULK OF AN 
INITIAL DETERMINATION FINDING NO VIOLATION OF §337 OF THE TARIFF ACT OF 
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1930. A copy of the initial determination and the commission determination are attached hereto 
as Exhibits 7 and 8 to the reissue appHcation. In addition, certain pleadings in each of these 
actions are included for consideration as Exhibits 4, 5 and 6. 

This reissue application is undertaken to make explicit what was previously impUcit in a 
proper construction of the claim language. Furthermore, certain erroneous claim construction 
findings by the Administrative Law Judge of the ITC are clear in the attached Initial 
Determination. For example, the limitation "a first port" found in claim 13 does not suggest or 
imply any decoding of addresses. 

The two year time period for submitting broadened claims has past, and it is believed that 
no broadening of any aspect of the claims is involved in this reissue application. 

AppUcant respectfully requests examination of the reissue apphcation at this regardless of 
the pending litigation. 

A Request for Reexamination is filed concurrently herewith. Applicant respectfully 
requests merger of this reissue appHcation with the Reexamination, should a determination be 
made that a substantial new question to patentability exists. See MPEP 1442.02, last paragraph. 

Please note that the documents provided fi-om the International Trade Commission action 
include only the public versions of the documents, certain information having been redacted in 
accordance with normal ITC practice. The undersigned was not privy to the ITC action and does 
not have access to the unredacted versions. 

The following documents are included as part of this submission: 

1 . A Reissue Application incuding a single column cut and paste version of specification 
of the patent sought to be reissued including changes made to the specification by two 
certificates of correction prior to this filing. Also, a cut and paste version of the claims 
and copies of the original drawings are included. 

2. Certificate under 37 C.F.R. 3.73(b)-unsigned (Exhibit 1 to the Reissue Application). 

3. Assignee's Consent to Reissue-unsigned (Exhibit 2). 

4. Offer to Surrender Letters Patent-unsigned (Exhibit 3). 

5. Complaint for Patent Infiingement in Cirrus Logic, Inc. vs. ATI Technologies, Inc. 
from the United States District Court, Northern District of California, San Francisco, 
Docket No. C-98-2700SI (Exhibit 4). 

6. Complaint under Section 337 of the Tariff Act of 1930 as amended (Exhibit 5). 
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7. First amended complaint under Section 337 of the Tariff Act of 1 930, as amended 
(Exhibit 6). 

8. PubHc version of the initial determination in the United States International Trade 
Commission investigation No. 337-TA-412, in the matter of certain video graphics display 
controllers and products containing same (Exhibit 7). 

9. Notice of Commission Determination not to review the bulk of an initial 
determination finding no violation of Section 337 of the Tariff Act of 1930 (Exhibit 8). 

10. Citation of Documents under 37 C.F.R. 1.555 together with 21 documents for 
consideration by the Patent and Trademark Office. 

Applicant respectfully requests that any unsigned document be provisionally accepted 
under 35 U.S. C. 26 pending submission of a signed document. 

Accordingly, Apphcant respectfully requests that the U.S. Patent & Trademark Office 
reissue U.S. Patent No. 5,598,525. 



Respectfully submitted, 
MCDERMOTT, WILL & EMERY 




David L. Stewart 
Registration No. 37,578 

600 13*^ Street, N.W. 
Washington, DC 20005-3096 
(202)756-8601 DLS:DLS 
Date: August 13, 1999 
Facsimile: (202) 756-8087 



